Commodity recognition apparatus

ABSTRACT

According to an embodiment, a commodity recognition apparatus includes a first camera configured to image a commodity, a posture sensor configured to acquire information indicating a posture of an operator, and a processor configured to acquire the image of the commodity from the first camera, execute commodity recognition processing on the image of the commodity to identify possible matches to previously registered commodity identities upon determining that the posture of the operator satisfies a specified condition according to information from the posture sensor, cause a plurality of commodity candidates are identified as possible matches in the commodity recognition processing, to be displayed on a display screen, and wait for an operator selection of one commodity candidate from the plurality of commodity candidates displayed on the display screen.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/857,126, filed Dec. 28, 2017, which application is based upon andclaims the benefit of priority from Japanese Patent Application No.2017-005359, filed Jan. 16, 2017, the entire contents of which areincorporated herein by reference.

FIELD

Embodiments described herein relate generally to a commodity recognitionapparatus.

BACKGROUND

In an existing commodity recognition apparatus, a commodity isrecognized based on an image of a commodity obtained with a scanner or acamera while the commodity is held by an operator. If an object isdetected, the commodity recognition apparatus executes recognitionprocessing to identify possible matches to previously registeredcommodity identities. There may be more than one possible match that isidentified in the recognition processing. If such is the case, thecommodity recognition apparatus displays the commodity candidates forselection of a particular candidate by the operator.

However, a conventional commodity recognition apparatus will sometimesdetect an object that is not actually a commodity to be identified andattempt the commodity recognition processing on this detected object.For example, if an arm or a shoulder of the operator unintentionallyappears in the scanner imaging window, the commodity recognitionapparatus could detect this as a possible commodity then start thecommodity recognition processing unnecessarily. If the commodityrecognition apparatus executes the commodity recognition processingbased on an image of an unintended object, the commodity recognitionapparatus might display match candidates and delay further processingfor a selection input by the operator. If the commodity recognitionapparatus waits for the selection of a candidate by the operator, theoperator will have to perform an operation for clearing the erroneouslydisplayed candidates, after which the operator will have tore-scan/re-image the commodity again. Such operations may lowerefficiency of the commodity registration.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a sales transaction registration system includinga commodity recognition apparatus according to an embodiment.

FIG. 2 is a block diagram of a commodity recognition apparatus and a POSterminal.

FIG. 3 is a schematic flowchart of accounting processing in a salestransaction registration system including a commodity recognitionapparatus.

FIG. 4 is a flowchart of a first operation example in a commodityrecognition apparatus.

FIG. 5 is a diagram of a plurality of commodity candidates in acommodity recognition apparatus.

FIG. 6 is a flowchart of a second operation example in a commodityrecognition apparatus.

DETAILED DESCRIPTION

In general, according to one embodiment, a commodity recognitionapparatus includes a first camera configured to image a commodity, aposture sensor configured to acquire information indicating a posture ofan operator, and a processor configured to acquire the image of thecommodity from the first camera, execute commodity recognitionprocessing on the image of the commodity to identify possible matches topreviously registered commodity identities upon determining that theposture of the operator satisfies a specified condition according toinformation from the posture sensor, cause a plurality of commoditycandidates that are identified as possible matches in the commodityrecognition processing, to be displayed on a display screen, and waitfor an operator selection of one commodity candidate from the pluralityof commodity candidates displayed on the display screen.

Hereinafter, an example of a sales transaction registration systemincluding a commodity recognition apparatus will be explained withreference to the drawings.

FIG. 1 is an exterior view of a sales transaction registration systemincluding a commodity recognition apparatus 1 according to anembodiment.

The sales transaction registration system includes the commodityrecognition apparatus 1 and a point-of-sale (POS) terminal 2. Thecommodity recognition apparatus 1 recognizes and registers a commoditybeing purchased by a customer. The POS terminal 2 performs settlementprocessing according to a price for the commodity recognized andregistered by the commodity recognition apparatus 1. The salestransaction registration system is, for example, a system that performsaccounting processing for a commodity purchased by a customer in a storesuch as a retail store. In the following explanation, it is assumed astore clerk, also referred to as a cashier, acts as an operator of thesales transaction registration system and carries out sales transactionprocessing including commodity recognition processing. However, in otherembodiments, the sales transaction registration system may be aself-service system in which a customer acts as an operator and carriesout the sales transaction processing including the commodity recognitionprocessing.

In the configuration example shown in FIG. 1, the commodity recognitionapparatus 1 is attached on a checkout counter 3. The commodityrecognition apparatus 1 includes a housing 1A, a keyboard 11, a touchpanel 12, and a customer display 13. The POS terminal 2 is on a registertable 4 via a drawer 5. The POS terminal 2 includes a keyboard 21, anoperator display 22, a customer display 23, and a receipt printer 24.The commodity recognition apparatus 1 and the POS terminal 2 areelectrically connected via a communication cable 400 (see FIG. 2).

The checkout counter 3, on which the commodity recognition apparatus 1is set, has an elongated shape generally paralleling a passage waythrough which the customer passes. The register table 4, on which thePOS terminal 2 is set, is placed on a downstream side with respect to amoving direction of the customer along the checkout counter 3. Theregister table 4 is placed substantially perpendicularly to the checkoutcounter 3. The checkout counter 3 and the register table 4 are spacedsuch that the store clerk acting as the operator can perform workin-between.

In the commodity recognition apparatus 1, the keyboard 11, the touchpanel 12, and the customer display 13 are attached to the housing 1A.The housing 1A houses a computer that functions as a control unit andperforms the commodity recognition processing. Further, the housing 1Aincludes a camera 14, a buzzer 15, and a posture sensor 16. In theconfiguration example shown in FIG. 1, the housing 1A has a thinrectangular shape and is erected substantially from the center of thecheckout counter 3. The keyboard 11 and the touch panel 12 arerespectively attached to the housing 1A with operation surfaces thereofdirected to the store clerk acting as the operator. The customer display13 is attached to the housing 1A with a display surface directed to thepassage side on which the customer passes.

The housing 1A includes a reading window 1B. The reading window 1B is anopening through which the camera 14 in the housing 1A photographs acommodity. In the configuration example shown in FIG. 1, the readingwindow 1B is formed in a rectangular shape on the operator side of thehousing 1A. The camera 14 includes an imaging sensor, a driving circuit,an optical system, and the like. The imaging sensor is, for example, aCCD (Charge Coupled Device) array. The driving circuit drives theimaging sensor. The optical system includes an imaging lens for formingan image of an imaging region photographed from the reading window 1B onthe CCD imaging element. The imaging region corresponds to a region of aframe image formed in an area of the CCD imaging element from thereading window 1B through the imaging lens. The camera 14 outputs theimage of the imaging region formed on the CCD imaging element.

Note that the camera 14 is not limited to particular configurationexplained above. Any camera that acquires, as image data, an imageincluding a commodity can be used for commodity recognition purposes.

In the configuration shown in FIG. 1, a basket 6 and a basket 7 aredisposed on either sides of the housing 1A on the checkout counter 3.Commodities that have not been through the commodity recognitionprocessing (referred to as unrecognized commodities) are stored in thebasket 6. Commodities that have been through the commodity recognitionprocessing (referred to as registered commodities) are put in the basket7. The basket 6 is placed on an upstream side of the housing 1A, withupstream and downstream being defined here with respect to ananticipated moving direction of the customer along the checkout counter3. The basket 7 is placed on a downstream side of the housing 1A.

For example, the customer places a basket 6, in which unrecognizedcommodities M that the customer desires to purchase are stored, on thecheckout counter 3. The operator takes out the commodities one by onefrom the basket 6 and holds up each over the reading window 1B. Inaccordance with an instruction by the operator, the camera 14 acquiresan image for each commodity through the reading window 1B. After thecommodity recognition processing on the image acquired by the camera 14is completed, the operator puts the now-recognized commodity in a basket7.

The posture sensor 16 is a sensor that detects information forestimating a position of the operator. Any sensor that acquires theposition information of the operator so as to permit a determination asto whether the operator, or a customer in the case of a self-serviceregister, is presently holding a commodity over the reading window 1Bcan be used as the posture sensor 16. For example, the posture sensor 16may be a camera, may be a distance measurement sensor, or may be aproximity sensor. The camera 14 may also be used as the posture sensor16.

FIG. 2 is a block diagram of the commodity recognition apparatus 1 andthe POS terminal 2.

The commodity recognition apparatus 1 includes a processor 111, a ROM(Read Only Memory) 112, a RAM (Random Access Memory) 113, an imageinterface 114, a device interface 115, a connection interface 116, and asensor interface 117. The processor 111 is connected to the ROM 112, theRAM 113, and the interfaces 114, 115, 116, and 117 via a bus line 118such as an address bus or a data bus.

The processor 111 executes a computer program, stored for example in ROM112, by which the processor 111 realizes various kinds of operations andfunctions. The processor 111 is, for example, a CPU (Central ProcessingUnit). The ROM (Read Only Memory) 112 stores computer programs andcontrol data. The RAM 113 is a volatile memory that stores data as awork area. The RAM 113 stores data that the processor 111 is processingor manipulating according to execution of a computer program.

The image interface 114 is an interface component that connects to thecamera 14. The processor 111 captures, via the image interface 114,image data photographed by the camera 14. The processor 111 stores, inthe RAM 113, the image data from the camera 14.

The device interface 115 is an interface component that connects to thekeyboard 11, the touch panel 12, and the customer display 13. The touchpanel 12 includes a display unit 12 a and a touch sensor 12 b. The touchsensor 12 b is disposed on a screen of the display unit 12 a. The deviceinterface 115 may also connect to the buzzer 15. The buzzer 15 emits,for example, an alarm sound. The buzzer 15 can receive a controlinstruction from the processor 111 via the device interface 115 causingan alarm to sound.

The connection interface 116 is an interface component that connects tothe POS terminal 2. In the configuration shown in FIG. 2, the connectioninterface 116 is connected to the POS terminal 2 via the communicationcable 400. Any interface method that permits communication with the POSterminal 2 can be used as the connection interface 116. For example, aninterface for wireless communication can be used as the connectioninterface 116.

The sensor interface 117 is an interface component that connects to theposture sensor 16. The sensor interface 117 complies with thecommunication requirements of the posture sensor 16. If the camera 14 isused as the posture sensor 16, the sensor interface 117 may be omittedas a separate component.

The POS terminal 2 includes, as shown in FIG. 2, a processor 211, a ROM212, a RAM 213, a HDD (Hard Disk Drive) 214, a device interface 215, acommunication interface 216, and a connection interface 217. Theprocessor 211 is connected to the ROM 212, the RAM 213, the HDD 214, andthe interfaces 215, 216, and 217 via a bus line 218, such as an addressbus or a data bus.

The processor 211 executes a computer program. For example, theprocessor 211 realizes various kinds of processing by executing computerprograms stored by the ROM 212. The processor 211 is, for example, a CPU(Central Processing Unit). The ROM 212 stores computer programs, controldata, and the like. The RAM 213 is a volatile memory that stores datafor use as a work area. The RAM 213 stores data that the processor 211processing or manipulating according to a control program being executedby the processor 211. The HDD 214 is a data storage unit. The datastorage unit is not limited to the HDD and may be, for example, a SSD(Solid State Drive).

The device interface 215 is an interface component that connects to thekeyboard 21, the operator display 22, the customer display 23, theprinter 24, and the drawer 5.

The communication interface 216 is an interface component that connectsto a store computer 300. The communication interface 216 communicateswith the store computer 300 via a network such as a LAN (Local AreaNetwork). The processor 211 is capable of transmitting and receivingdata to and from the store computer 300 via the communication interface216. For example, the processor 211 receives a dictionary file 500 forcommodity recognition from the store computer 300 via the communicationinterface 216. The processor 211 saves the dictionary file 500 receivedfrom the store computer 300 in the HDD 214.

The connection interface 217 is an interface component that permitscommunication with the commodity recognition apparatus 1. In theconfiguration example shown in FIG. 2, the connection interface 217 isconnected to the connection interface 116 of the commodity recognitionapparatus 1 via the communication cable 400. For example, the processor211 controls display of the touch panel 12 and the customer display 13of a commodity recognition apparatus 1 via the connection interface 217or the like. The processor 111 of the commodity recognition apparatus 1accesses the dictionary file 500 in the HDD 214 of the POS terminal 2through communication via the connection interface 217 or the like.

The dictionary file 500 includes data for identifying each commoditybeing sold at the store. For example, the dictionary file 500 includesreference image data and feature value data associated with eachcommodity ID and a corresponding commodity name of each commodity. Thereference image data can be image data obtained by photographing theparticular commodity associated with each commodity ID under variousconditions. For example, the reference image data can be obtained byphotographing the commodity from various directions. The feature valuedata is parametric data obtained by analysis of feature values inreference images. The feature value data is collated with feature valuesin a commodity images used in commodity recognition processing.

Processing functions in the commodity recognition apparatus 1 will beexplained below.

The commodity recognition apparatus 1 has various processing functionsthat are realized by the processor 111 executing a computer program orprograms. The processor 111 executes the computer programs stored in theROM 112, whereby the commodity recognition apparatus 1 realizes variousprocessing functions such as commodity registration processing includingthe commodity recognition processing. For example, the commodityrecognition apparatus 1 has an image acquiring function, an imagedetecting function, a feature-value extracting function, asimilarity-degree calculating function, a commodity specifying function,and a result output function as processing functions for recognizing acommodity.

In the commodity recognition apparatus 1 according to this exampleembodiment, object recognition is adopted as a technique for commodityrecognition. The object recognition is a technique for recognizing, fromimage data of a photographed image including an object, a type orcategory of the object. For example, the object recognition is realizedby a processor in a computer executing a computer program. The imagedata is acquired and feature values for recognition an object includedin the acquired image data are extracted. The degrees of similarity arethen calculated using feature values extracted from the image andfeature value data already recorded as dictionary data. The objectrecognition is performed in accordance with the calculated degree(s) ofsimilarity.

The image acquiring function is a function of acquiring an image fromthe camera 14. The camera 14 photographs a commodity held in front ofthe reading window 1B to obtain an image. The processor 111 acquires theimage via the image interface 114.

The image detecting function is a function of detecting a commodityimage in the image from the camera 14. The processor 111 detects animage region corresponding to the commodity in the image from the camera14. The processor 111 segments the image region of the commodity andsets the image region as a commodity image.

The feature-value extracting function is a function of extracting afeature value from the commodity image. The processor 111 extractsfeature values of the commodity using an algorithm adopted for commodityrecognition processing.

The similarity-degree calculating function is a function of calculatingsimilarity degrees by reference to the feature values extracted from thecommodity image and the feature value data of the commodities registeredas dictionary data. The processor 111 calculates similarity degrees ofthe feature values obtained from the commodity image and the featurevalue data in the dictionary file 500 for the commodities.

The commodity specifying function is a function of specifying thecommodity of the image from the camera 14 in accordance with thesimilarity-degree calculating function based on feature values. Theprocessor 111 identifies the commodity in the commodity image inaccordance with the similarity degrees or, alternatively, selectsmultiple commodity candidates. For example, if there is only onecommodity having a similarity degree equal to or larger than somepredetermined threshold value, the processor 111 specifies the commodityin is this one commodity in the dictionary file 500 meeting or exceedingthe threshold value. If there is no commodity in the dictionary filehaving a similarity degree equal to or greater than the predeterminedthreshold or if there is a plurality of commodities having similaritydegrees equal to or greater than the predetermined threshold, theprocessor 111 selects multiple commodity candidates in a descendingorder of the similarity degrees.

If a plurality of commodity candidates is identified and provided, theoperator is required to select one commodity from the candidates tospecify the commodity in the commodity image.

The result output function is a function of outputting informationindicating the commodity specified by the commodity specifying functionto the POS terminal 2. If the commodity is specified by the commodityspecifying function, the processor 111 outputs a commodity ID of thespecified commodity to the POS terminal 2 as matching the commodityphotographed by the camera 14.

The processor 111 may include a code recognizing function forrecognizing a code (e.g., a barcode or a two-dimensional code) forspecifying a commodity from the commodity image. If the code can bedetected in the commodity image, the processor 111 may recognize acommodity associated with the code. If the code cannot be detected, theprocessor 111 may recognize a commodity according to the objectrecognition. However, in the following explanation, it is assumed thatthe commodity recognition apparatus 1 recognizes the commodity accordingto the object recognition processing.

A flow of accounting processing in the sales transaction registrationsystem will be explained below.

FIG. 3 is a schematic flowchart of the accounting processing in thesales transaction registration system.

As shown in FIG. 3, the sales transaction registration system performscommodity registration processing for registering commodities purchasedby the user and accounting processing for settling prices of all thecommodities registered by the commodity registration processing. Thecommodity registration processing includes specifying individualcommodities purchased by the user or the customer according to commodityrecognition by the objet recognition.

A customer searches for commodities to purchase in the store and putscommodities to be purchased in the basket 6. The customer carries thebasket 6 to the checkout counter 3. The customer or the store clerkplaces the basket 6 on the near side of the housing 1A on the upstreamside of the checkout counter 3. The operator, who can be the store clerkor the user, takes out the commodities from the basket 6 one by one andholds the commodity over the reading window 1B. If the recognition ofthe commodity held over the reading window 1B is successful, theoperator puts the commodity in the basket 7.

The commodity recognition apparatus 1 registers a commodity specified bythe commodity recognition processing for an image photographed by thecamera 14 (ACT 11). The commodity recognition apparatus 1 photographs,with the camera 14, images of individual commodities held over thereading window 1B. The commodity recognition apparatus 1 performs thecommodity recognition processing on the images photographed by thecamera 14. In the commodity recognition processing, the commodityrecognition apparatus 1 also acquires posture information for estimatinga posture or position of the operator from the posture sensor 16 andmonitors the position of the operator. If determining, by reference tothe posture information, that the posture of the operator allows thecommodity recognition processing to be executed, the commodityrecognition apparatus 1 continuously carries out the commodityrecognition processing for the commodities.

The commodity registration processing including the commodityrecognition processing will be explained in detail below. It is assumedthat the processor 111 carries out the commodity registration processand supplies information indicating the registered commodities to thePOS terminal 2. However, the processor 211 of the POS terminal 2 mayinstead carry out the commodity recognition processing.

If the commodity registration processing is completed and allcommodities are registered, the operator instructs registrationcompletion with the key board 11 or the touch panel 12 of the imagerecognition apparatus 1. That is, after the commodity registrationprocessing is started, the commodity recognition apparatus 1 receivesthe instruction for the registration completion of the commodities bythe operator (ACT 12). If the registration completion of the commoditiesis not instructed (NO in ACT 12), the commodity recognition apparatus 1continues to execute the commodity registration processing.

If the registration completion is instructed (YES in ACT 12), thecommodity recognition apparatus 1 starts the accounting processing bythe POS terminal 2 for the registered commodities (ACT 13). For example,the processor 211 acquires commodity IDs of the registered commoditiesfrom the commodity recognition apparatus 1. If the processor 211acquires the commodity IDs of the registered commodities, the processor211 then specifies commodity prices of the commodities in accordancewith the acquired commodity IDs.

The processor 211 calculates a settlement amount for all the commoditiesthat have been registered by totaling the commodity prices of theregistered commodities. After calculating the settlement amount, cash orpayment for settling the calculated settlement amount is then receivedfrom the customer. After receiving notification of full payment by thecustomer, the processor 211 issues a receipt indicating transactiondetails and ends the sales transaction processing.

A first operation example of the commodity registration processing inthe commodity recognition apparatus 1 will be explained below.

FIG. 4 is a flowchart of the first operation example of the commodityregistration processing including the commodity recognition processingin the commodity recognition apparatus 1.

In the first operation example, first, the processor 111 of thecommodity recognition apparatus 1 clears display content of the touchpanel 12 (ACT 31) and starts the commodity registration processing.After starting the commodity registration processing, the processor 111acquires information indicating a posture or position of the operator,also referred to as posture information, from the posture sensor 16 (ACT32). After acquiring the posture information from the posture sensor 16,the processor 111 determines in accordance with the acquired postureinformation whether the operator is in a posture that allows thecommodity recognition processing to be executed (ACT 33). The processor111 determines, according to whether the posture information acquiredfrom the posture sensor 16 satisfies a previously specified condition,whether the operator is in the posture that allows the commodityrecognition processing to be executed.

If the posture sensor 16 is a sensor that acquires body frame orposition information, the processor 111 estimates the posture of theoperator according to the body frame or position information serving asthe posture information acquired by the posture sensor 16. If theposture sensor 16 is a camera, the processor 111 estimates the postureof the operator from an image photographed by the camera functioning asthe posture sensor 16. If the posture sensor 16 is a proximity sensor,the processor 111 estimates the posture or the position of the operatoraccording to a detection signal of the proximity sensor functioning asthe posture sensor 16.

In the first operation example, it is assumed that the posture thatallows the commodity recognition processing to be executed is a posturein which the operator is holding or can hold a commodity over thereading window 1B. For example, the posture that allows the commodityrecognition processing to be executed may be postures such as an“upright state”, “the position of the head is within a predeterminedrange”, and “the face is facing front towards the camera 14”. A state inwhich the operator is not close to the proximity sensor functioning asthe posture sensor 16 provided in a predetermined position (e.g., abovethe touch panel 12) may be the posture that allows the commodityrecognition processing to be executed.

For example, the processor 111 can determine according to the postureinformation (e.g., the body position information) acquired by theposture sensor 16 whether the operator is in an upright state. If acondition that the posture information indicates the upright state ofthe operator is set as the specified condition, the processor 111determines according to the posture information acquired by the posturesensor 16 whether the operator is in the upright posture.

If the posture sensor 16 is a camera, the processor 111 can estimate aposition of the head of the operator from an image photographed by thecamera functioning as the posture sensor 16 (If an allowable range forthe position of the head is set as the specified condition, theprocessor 111 determines whether the position of the head specified fromthe posture information is within the allowable range.

If the posture sensor 16 is a camera, the processor 111 can estimate adirection of the face of the operator from the image photographed by thecamera serving as the posture sensor 16. If the direction of the face isset as the specified condition, the processor 111 determines whether thedirection of the face specified from the posture information is thedirection set as the specified condition.

If determining that the operator is in the posture that allows thecommodity recognition processing to be executed (YES in ACT 34), theprocessor 111 acquires a photographed image from the camera 14 with theimage acquiring function (ACT 35). That is, the processor 111 captures aphotographed image as a frame image from the camera 14 via the imageinterface 114. The processor 111 stores the photographed image capturedvia the image interface 114 in the RAM 113.

The processor 111 performs processing for detecting a commodity image inthe photographed image acquired from the camera 14 (ACT 36). Theprocessor 111 detects a region of an object estimated as a commodityfrom the photographed image stored in the RAM 113 to thereby determinewhether an object is present in the photographed image. For example, theprocessor 111 extracts or detects a contour line or the like from animage obtained by binarizing the photographed image. The processor 111attempts extraction of a contour of the object estimated as thecommodity from the binarized image. After extracting the contour of theobject, the processor 111 regards the inside of the extracted contour asan image region, referred to as a commodity image, of the commodity.

If determining that an object is absent in the photographed image, thatis, if a commodity image cannot be extracted from the photographed image(NO in ACT 36), the processor 111 returns to ACT 32. In this case, whenthe posture information acquired from the posture sensor 16 satisfiesthe specified condition, the processor 111 captures the nextphotographed image from the camera 14. That is, when the postureinformation satisfies the specified condition, the processor 111sequentially captures photographed images from the camera 14 until acommodity image can be extracted. Every time the processor 111 capturesthe photographed image from the camera 14, the processor 111 executesextraction processing for a commodity image. Therefore, if the commodityis held over the reading window 1B such that the posture informationsatisfies the specified condition, the processor 111 extracts acommodity image from the photographed image of the camera 14 and startsthe commodity recognition processing.

If determining that an object is present in the photographed image, thatis, if a commodity image can be extracted from the photographed image(YES in ACT 36), the processor 111 executes the commodity recognitionprocessing for the commodity image (ACT 37).

In the commodity recognition processing, the processor 111 recognizes acommodity with the object recognition by the feature-value extractingfunction, the similarity-degree calculating function, and the commodityspecifying function. That is, the processor 111 extracts, with thefeature-value extracting function, feature values used for the objectrecognition from the commodity image. After extracting the featurevalues from the commodity image, the processor 111 calculates, with thesimilarity-degree calculating function, similarity degrees of theextracted feature values and the feature value data of the dictionaryfile 500 of the commodities. The processor 111 determines whether thesimilarity degrees for the commodities calculated by the commodityspecifying function are equal to or larger than the threshold.

If there is one commodity having a similarity degree equal to or largerthan the threshold, that is, if only one commodity having sufficientlyhigh accuracy of a recognition result is obtained, the processor 111specifies that a commodity of the commodity image is the commodityhaving the similarity degree equal to or larger than the threshold (YESin ACT 38). If a commodity is specified as one commodity by thecommodity recognition processing, the processor 111 registers thespecified commodity as a settlement target commodity without involvingoperation by the operator (ACT 42). That is, if a commodity can beuniquely specified by the commodity recognition processing, theprocessor 111 automatically registers the specified commodity withoutinvolving operation by the operator. This function may be referred to asan automatic decision function.

If there is no commodity having a similarity degree equal to or largerthan the threshold (NO in ACT 38), the processor 111 selects a pluralityof commodity candidates in the descending order of similarity degrees.If there are a plurality of commodities having similarity degrees equalto or larger than the threshold (NO in ACT 38), the processor 111selects the plurality of commodities having the similarity degrees equalto or larger than the threshold as commodity candidates. After selectingthe plurality of commodity candidates, the processor 111 displays theselected plurality of commodity candidates on the touch panel 12 (ACT39).

FIG. 5 is a display example including a plurality of commoditycandidates obtained in the commodity recognition processing.

In the display example shown in FIG. 5, the touch panel 12 displays acommodity image and displays a plurality of icons N1 to N4 indicating aplurality of commodity candidates. The icons N1 to N4 are buttonsselectable by the touch sensor 12 b of the touch panel 12. The icons N1to N4 respectively indicate the commodity candidates. The processor 111detects a touch on any one of the icons N1 to N4.

If the plurality of commodity candidates is selected, the processor 111displays the selectable commodity candidates as shown in FIG. 5 tothereby receive selection of a commodity candidate by the operator (ACT40). The processor 111 displays the selectable commodity candidates andreceives an instruction for reset of the commodity candidates, orretrial of the commodity recognition processing, by the keyboard 11 orthe touch panel 12 (ACT 41). If the reset of the commodity candidates isinstructed (YES in ACT 41), the processor 111 returns to ACT 31, clearsthe commodity candidates, and executes the processing explained aboveagain. If the operator selects any one of the commodity candidates (YESin ACT 40), the processor 111 registers the commodity candidate selectedby the operator as a settlement target commodity (ACT 42).

If the plurality of commodity candidates is displayed, the processor 111maintains the display until the operator selects any one of thecommodity candidates or the operator instructs to reset. Whilemaintaining the display, the processor 111 stops the commodityrecognition processing. However, the processor 111 may repeatedly carryout the processing in ACTS 35 to 38 while maintaining the display of theplurality of commodity candidates. In this case, if one commodity havinga similarity degree equal to or larger than the threshold can bespecified, the processor 111 may register, with the automatic decisionfunction, the commodity having the similarity degree equal to or largerthan the threshold.

In the first operation example, the commodity recognition apparatusdetermines in accordance with the posture information acquired by theposture sensor whether the operator is in the posture that allows thecommodity recognition processing to be executed. If determining,according to the posture information, that the operator is in theposture that allows the commodity recognition processing to be executed,the commodity recognition apparatus executes the commodity recognitionprocessing.

Consequently, the commodity recognition apparatus can prevent thecommodity recognition processing unintentionally being executed when anobject other than a commodity such as an arm or a shoulder of theoperator approaches the reading window. For example, when the operatortakes commodities out from a basket, even if the arm, the shoulder, orthe like unintentionally approaches the reading window or the camera, itis possible to prevent the commodity recognition processing from beingexecuted. As a result, unnecessary commodity candidates are notdisplayed by the commodity recognition processing for an unintendedphotographed image (a photographed image of an object other than acommodity). Useless reset operation and the like are unnecessary.

A second operation example of the commodity registration processing inthe commodity recognition apparatus 1 will be explained.

FIG. 6 is a flowchart of the second operation example of the commodityregistration processing including the commodity recognition processingin the commodity recognition apparatus 1.

In the second operation example, first, the processor 111 of thecommodity recognition apparatus 1 defines a posture flag P and an updateflag U. For example, the processor 111 sets the posture flag P and theupdate flag U on the RAM 113. After setting the posture flag P and theupdate flag U, the processor 111 sets an initial value of the postureflag P to 1 (ACT 50) and sets an initial value of the update flag U to 0(ACT 51).

After setting the posture flag P and the update flag U to the initialvalues, the processor 111 clears display content of the touch panel 12(ACT 52) and starts the commodity registration processing. Afterstarting the commodity registration processing, the processor 111acquires a photographed image with the image acquiring function (ACT53). That is, the processor 111 captures, via the image interface 114, aphotographed image as a frame image from the camera 14. The processor111 stores the photographed image captured via the image interface 114in the RAM 113.

The processor 111 performs processing for detecting a commodity image inthe photographed image acquired from the camera 14 using the imagedetecting function (ACT 54). The processor 111 detects a region of anobject estimated as a commodity from the photographed image stored inthe RAM 113 to thereby determine whether an object is present in thephotographed image.

If determining that an object is absent in the photographed image, thatis, if a commodity image cannot be extracted from the photographed image(NO in ACT 53), the processor 111 returns to ACT 53 and acquires thenext photographed image. That is, the processor 111 sequentiallycaptures photographed images as frame images until a commodity image canbe extracted. Every time the processor 111 captures a photographed imagefrom the camera 14, the processor 111 executes the extraction processingfor a commodity image.

If determining that an object is present in the photographed image, thatis, if a commodity image can be extracted from the photographed image(YES in ACT 54), the processor 111 executes the commodity recognitionprocessing for the commodity image (ACT 55). The processor 111 extractsfeature values used for the object recognition from the commodity image.The processor 111 calculates similarity degrees of the extracted featurevalues and the feature value data of the dictionary file 500 of thecommodities. The processor 111 determines whether the calculatedsimilarity degrees for the commodities are equal to or larger than thethreshold.

If the processor 111 specifies one commodity having a similarity degreeequal to or larger than the threshold (YES in ACT 56), the processor 111registers the commodity having the similarity degree equal to or largerthan the threshold as a settlement target commodity (ACT 59).

If there is no commodity having a similarity degree equal to or largerthan the threshold (NO in ACT 56), the processor 111 selects a pluralityof commodity candidates in the descending order of the similaritydegrees. If there is a plurality of commodities having similaritydegrees equal to or larger than the threshold (NO in ACT 56), theprocessor 111 selects the plurality of commodities having the similaritydegrees equal to or larger than the threshold as commodity candidates.After selecting the plurality of commodity candidates, the processor 111displays the selected plurality of commodity candidates on the touchpanel 12 (ACT 57).

The processor 111 receives selection of a commodity candidate by theoperator by displaying the plurality of commodity candidates in aselectable state. If the operator selects any one of the commoditycandidates (YES in ACT 58), the processor 111 registers the commodityselected by the operator as a settlement target commodity (ACT 59).

The processor 111 displays the plurality of commodity candidates andreceives an instruction to reset the displayed commodity candidates. Ifthe reset of the commodity candidates is instructed or if the updateflag U is 1 (YES in ACT 60), the processor 111 returns to ACT 51, clearsthe commodity candidates, and executes the processing explained aboveagain.

Further, the processor 111 displays the plurality of commoditycandidates and monitors a posture state of the operator (ACTS 61 to 63).That is, the processor 111 acquires information indicating the postureof the operator, also referred to as posture information, from theposture sensor 16 (ACT 61). After acquiring the posture information, theprocessor 111 determines in accordance with the acquired postureinformation whether the operator is in a posture permitting successfulexecution of the commodity recognition processing (ACT 62). Theprocessor 111 determines, according to whether the posture informationacquired by the posture sensor 16 satisfies a specified condition,whether the operator is in the posture that will likely allow thecommodity recognition processing to be executed successfully. Processingin ACT 62 may be the same processing as ACT 33 explained above.

If determining that the operator is not in the posture that allows thecommodity recognition processing to be executed (posture NG)successfully (NO in ACT 63), the processor 111 sets the posture flag Pto 0 (ACT 64) and returns to ACT 57.

If determining that the operator is in the posture that allowssuccessful commodity recognition processing to be executed (posture OK)(YES in ACT 63), the processor 111 determines whether the posture flag Pis 0 (ACT 65). If the posture flag P is not 0 (NO in ACT 65), theprocessor 111 returns to ACT 57 without changing the posture flag P andthe update flag U. If the posture flag P is 0 (YES in ACT 65), theprocessor 111 sets the posture flag P to 1 and sets the update flag U to1 (ACT 66) and returns to ACT 57.

In ACT 66, the processor 111 detects whether the posture of the operatortransitions from a state of posture NG (a first state) to a state ofposture OK (a second state). For example, if the upright state is thestate corresponding to the posture OK state, the update flag U changesto 1 (ON) when the posture of the operator transitions from anon-upright state to the upright state. If the update flag U changes to1, the processor 111 determines YES in ACT 60 and returns to ACT 51,clears the commodity candidates, and executes the commodity recognitionprocessing again.

In the second operation example, during the display of the commoditycandidates, the commodity recognition apparatus monitors, according tothe posture information, the transition of the operator's posture fromthe posture NG to the posture OK. If the posture of the operatortransitions from the posture NG to the posture OK, the commodityrecognition apparatus clears the displayed commodity candidates andexecutes the commodity recognition processing again.

Consequently, even if unnecessary commodity candidates are displayedafter commodity recognition processing on an unintended photographedimage, the unnecessary commodity candidates are cleared once theoperator takes a posture for holding a commodity for imagining. If theoperator takes the posture for holding a commodity for imagining, theprevious commodity candidates are cleared and the commodity recognitionprocessing is executed again to determine candidates according to anewly acquired image of the commodity. Therefore, it is possible tosmoothly carry out the commodity recognition processing on an intendedphotographed image by monitoring changes in the posture of the operator.

While certain embodiments have been described these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms. Furthermore variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the invention.

What is claimed is:
 1. A commodity recognition apparatus comprising: adisplay; a first camera configured to image a commodity; a posturesensor configured to detect a posture of an operator holding thecommodity over a reading window; and a processor configured to acquire,from the first camera, the image of the commodity that is held over thereading window, upon detecting that the posture of the operator is in anupright state, execute commodity recognition processing on the image ofthe commodity to identify one or more commodity candidates, and controlthe display to display the identified commodity candidates and then waitfor an input that selects one of the commodity candidates, wherein theposture sensor detects that the posture of the operator is in an uprightstate when the head of the operator is located within a predeterminedrange.
 2. The commodity recognition apparatus according to claim 1,wherein the processor executes the commodity recognition processing byperforming object recognition using a dictionary file.
 3. The commodityrecognition apparatus according to claim 2, further comprising: astorage device that stores the dictionary file.
 4. The commodityrecognition apparatus according to claim 1, wherein the posture sensoris a second camera.
 5. The commodity recognition apparatus according toclaim 1, wherein the first camera is also configured as the posturesensor.
 6. A commodity recognition apparatus comprising: a display; afirst camera configured to image a commodity; a posture sensorconfigured to detect a posture of an operator holding the commodity overa reading window; and a processor configured to acquire, from the firstcamera, the image of the commodity that is held over the reading window,upon detecting that the posture of the operator is in an upright state,execute commodity recognition processing on the image of the commodityto identify one or more commodity candidates, control the display todisplay the identified commodity candidates and then wait for an inputthat selects one of the commodity candidates, and during the waiting,upon detecting that the posture of the operator is not in an uprightstate, control the display not to display the identified commoditycandidates, acquire another image of the commodity from the firstcamera, and execute the commodity recognition processing on said anotherimage of the commodity.
 7. The commodity recognition apparatus accordingto claim 6, wherein the processor executes the commodity recognitionprocessing by performing object recognition using a dictionary file. 8.The commodity recognition apparatus according to claim 7, furthercomprising: a storage device that stores the dictionary file.
 9. Thecommodity recognition apparatus according to claim 6, wherein theposture sensor detects that the posture of the operator is in an uprightstate when the head of the operator is located within a predeterminedrange.
 10. The commodity recognition apparatus according to claim 6,wherein the posture sensor detects that the posture of the operator isin an upright state when the face of the operator is facing towards thefirst camera.
 11. The commodity recognition apparatus according to claim6, wherein the posture sensor is a second camera.
 12. The commodityrecognition apparatus according to claim 6, wherein the first camera isalso configured as the posture sensor.
 13. A method for commodityrecognition, the method comprising: acquiring an image of a commoditythat is held by an operator over a reading window; detecting a postureof the operator holding the commodity over the reading window; upondetecting that the posture of the operator is in an upright state,executing commodity recognition processing on the image of the commodityto identify one or more commodity candidates; and displaying theidentified commodity candidates and then waiting for an input thatselects one of the commodity candidates, wherein the posture of theoperator that is in an upright state is detected when the head of theoperator is located within a predetermined range.
 14. The methodaccording to claim 13, wherein executing commodity recognitionprocessing includes performing object recognition using a dictionaryfile.
 15. The method according to claim 14, further comprising: storingthe dictionary file in a storage device.
 16. The method according toclaim 13, wherein the image of the commodity and the first postureinformation are acquired by a camera.